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Fourth Interchapter 


Collecting and Working with 
Census Data 


Designing research for the future of the field of Writing Studies is crucial 
as student needs and demographics, and institutional demands, shift. 
In this complex and global context, the major takeaways for this inter- 
chapter relate to the use of census data collection and external sites to 
contextualize Writing Studies research. Here are some topics Pll discuss 
and illustrate with my own research process: 


e Soliciting and Collecting Institutional Review Board (IRB) Metadata 
e Contextual Data and Useful Census Data 
e Methodological Reflections 


Soliciting and Collecting IRB Metadata 


I initially grew curious about recruitment and retention during my time 
working as an IRB coordinator. I noticed, upon closing studies at the 
request of researchers, that many studies had not enrolled the quota 
the researcher had claimed was necessary to generalize to their target 
population or make qualified inferences based on the context of the 
research—I refer to these studies as under-enrolled. I was interested in 
what sort of recruitment methods researchers used and what their final 
enrollment counts were when they closed out their studies. In presenting 
this empirical metadata on IRBs and research, I focus on the closure 
of protocols approved under the expedited criteria, as these are proj- 
ects that underwent review by board members. Either individual board 
members or the full board approved these research studies per the active 
policy at the time, 45 CFR § 46.110. I removed exempt studies from the 
dataset, as these are determined to be “exempt” from the more com- 
prehensive demands of the expedited and full board criteria. Though 
exempt studies made up the majority of research filed by respondents 
in the IRBs and Writing Studies Survey, not all research we conduct is 
exempt. And this metadata is useful for all researchers. A glance at IRBs’ 
data helps us better understand the broader regulatory and institutional 
frameworks that impact research with participants. 

When researchers are ready to close studies that were approved by an 
IRB, researchers provide the number of total enrolled participants, as 
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well as information regarding publications, any unanticipated events, 
and other information of interest. I collected three pieces of information 
from study closures from the IRBs: (1) the number of expected partici- 
pants researchers reported in their IRB protocol when it was approved, 
(2) the final number of enrolled participants, and (3) method(s) for re- 
cruiting participants. 


IRB #1 


For IRB #1, collection of one year’s worth of closures ensured an equal 
distribution of closures to account for the ebbs and flows of the academic 
and calendar year with a variety of researchers. The collected closures 
only represent the studies reviewed and approved under expedited pro- 
cedures that were closed during that time period at the researchers’ re- 
quest, or due to a force closure (when a researcher is non-responsive to 
IRB #1, their studies are terminated). 

IRB #1 serves a mid-size, land-grant research university in the West- 
ern U.S. There are approximately 27,000 students enrolled (a little over 
3,000 of these are graduate students) and 800 faculty; in fiscal year 
2016, approximately 27% of the institution’s revenue was generated by 
contracts, grants, and federal appropriations. I collected data from this 
Institutional Review Board from March 1, 2016 to February 28, 2017. 

Because there is no medical school associated with IRB #1, the major- 
ity of research reviewed was minimal risk and approvable under expe- 
dited procedures—similar to some research with human participants in 
Writing Studies. During the same period of time, a substantial amount 
of research was determined exempt from the federal regulation by IRB 
#1. At this institution, like many IRBs that are accredited by bodies 
such as the Association for the Accreditation of Human Research Pro- 
tection Programs (AAHRPP), these determinations are made by IRB 
staff. Therefore, the IRB has experience implementing the Common 
Rule with disciplines in the social sciences and humanities, as well as ex- 
empting research from the regulations. At the time I was collecting data, 
I would suggest this IRB was adapting to the era of hypercompliance 
(Babb, 2020), not due to any malfeasance on the part of researchers, but 
because of a shift in leadership. 

For IRB #1, 127 protocols were closed by the IRB between March 1, 
2016 and February 28, 2017. Ninety-eight were used in this analysis. 
To prevent a significant outlier from heavily skewing the data, I gleaned 
the final 98 via screening, which is exhibited in the PRISMA diagram in 
Figure 4IC.1. 

The major outlier removed from IRB #1’s dataset had estimated a nec- 
essary enrollment of millions of participants and did not come close to 
their enrollment target. This protocol was for a federal service program 
and the lack of enrollment heavily skewed the original dataset. As an 
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Removed: cancelled studies (7), external review (2), expedited review, 1991 
policy category 5 (10), force closed/insufficient closure data (9), outlier (1) 


Studies included in analysis 
(n =98) 


Figure 4IC.1 PRISMA diagram moving from 128 initial closures to a final set 
of 98. 


outlier, it was removed for the purposes of this analysis. The remaining 
studies were removed to their categorization as cancelled, missing clo- 
sure data, or because their review type did not contribute to the dataset 
for these analytic purposes. With the final 7 = 98, the overall enrollment 
rate (the number of participants actually enrolled compared to the num- 
ber of participants required for generalization, per the researcher) for ex- 
pedited studies closed during the collection period at IRB #1 was 76%. 


IRB #2 


IRB #2 provided census data to me in a spreadsheet. Given the size of 
the IRB, Iam fortunate I did not have to manually collect data. This IRB 
is associated with a university hospital, medical school, and multiple 
clinics. IRB #2 runs a home-built, sophisticated protocol management 
database. I asked their IRB Director if it was possible to query the sys- 
tem for my specific target information: IRB approval type, proposed 
enrollment, recruitment methods, and final enrollments. They were able 
to provide closure data for approximately eight years, ending in early 
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September 2018. Because this data cleaning process was more complex, 
Pll discuss briefly how I arrived at my final dataset of the 1,720 protocols 
used in this analysis. 

The original spreadsheet had over 17,000 rows and each row was in- 
tended to represent a specific protocol. However, some errant html code 
was pulled in the data query and incorporated into the excel sheet. Ad- 
ditionally, protocols in the initial spreadsheet were missing information 
in one of the fields. I used Excel’s “sort” function to collect only those 
protocols with digits—not ranges of digits or alphabetic entries—in the 
“# Approved” and reported “Final Enrollment” columns. The removal 
of those studies with a range or alphabetic entries is important to note, 
because further cleaning of this data could present a different range of 
enrollment. However, given the sheer volume of entries for this IRB, re- 
moving these during the first cleaning step still resulted in a robust and 
representative dataset. 

This first sorting process also forced the erroneous lines of code to 
drop to the bottom of the sheet, and I was able to remove them. I re- 
moved retrospective studies and chart reviews, and cancelled studies 
(studies with 0 enrollment) as I did for IRB #1. I also removed all exempt 
protocols. What remained were 2,723 complete rows. When I ran initial 
calculations, it appeared that researchers who had their studies reviewed 
by IRB #2 had a dismal 15% participant enrollment rate. Again, these 
were researcher/principal investigator (PI) determined enrollment needs 
for studies to meet researcher’s self-stated generalizability criteria. This 
dataset also excluded studies occupying any row that did not report all 
data completely, as was required for this analysis. 

This ratio was astoundingly bad. I figured, as with IRB #1’s dataset, 
there might be an outlier. However, once I sorted the data again, I re- 
alized there were, in many instances, duplicates or triplicates of certain 
protocols in the dataset. It was clear that data had been pulled from 
these studies’ annual continuation reviews. I removed all duplicates and 
then, when a protocol was still duplicated but numbers in the “Final 
Enrolled” column were different, I removed those rows where the final 
enrolled amount was lowest (suggesting a study in progress). What re- 
mained was the final set of 1,720 closed protocols that were reviewed 
either by the convened IRB or with expedited procedures by IRB #2. In 
this final dataset, researchers at IRB #2’s institution had a 44.5% enroll- 
ment rate in studies closed over an eight-year period. In the same time 
frame, over 800 studies were closed that had a reported enrollment of 0. 


What the Data Indicate 


The data suggest that these two very different, yet high functioning, IRBs 
are overburdened reviewing research that fails to achieve the promise of 
generalizability that the researchers outline in their proposed projects. 
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Based on this snapshot of data, IRBs regularly review research that both 
fails to enroll sufficient participants while at the same time also capital- 
izing on participants’ goodwill without the promise of any contribution 
to the greater good. This, in turn, suggests researchers could be, in the 
worst-case scenario, designing poor research studies that involve par- 
ticipants with varying levels of risk without the possibility of benefit 
(violating the principle of beneficence). 

Situating these concerns, which I take up in Chapter 4, within broader 
realities regarding the populations we do and will serve in our evolving 
demographic contexts presents a bleak outlook. IRBs, with their focus 
on broader issues of beneficence and justice, in particular, can serve as 
helpful moderators to determine how valuable research can be for the 
communities it is intended to impact. Beyond understanding this context 
for IRBs’ review of research, it is imperative to be aware of other pressing 
concerns that IRBs increasingly find themselves addressing: specifically, 
the recruitment and engagement of diverse populations within research. 
Some IRBs take this work as part of their mission. Others recognize it as 
central but do not apply standards and defer instead to researchers. It is 
critical at this moment to have a disciplinary approach to diversity and 
equity—both matters of justice—as we design research and work with 
our IRBs. The next section provides context for the collection and use of 
further census data that will shape the generalizability and transferabil- 
ity of Writing Studies research with participants. 


Contextual Data and Useful Census Data 


In combing through larger sets of data for information about our future 
as a field, I turned to spaces like the U.S. Census and the National Center 
for Education Statistics. I rely heavily on the data generated by these 
organizations because the data clearly illustrate why diversity, effective 
recruitment, reproducibility, justice, and research design are such crucial 
spaces for discussion. 

National Center for Educational Statistics data, a unit of the Institute 
of Education Sciences and the Department of Education, offer clear in- 
dicators of how demographic changes will impact our classrooms and 
research in the coming years. Between 2017 and 2028, we can anticipate 
increased representation of the following race/ethnicities in postsecond- 
ary enrollments (Hussar & Bailey, 2020, p. 25): 


e Fourteen percent increase enrollment in Hispanic students (4 
million, up from 3.5 million) 

e Eight percent increase enrollment in Black students (2.7 million, up 
from 2.5 million) 

e Two percent increase in enrollment of Asian/Pacific Islander students 
(1.4 million, up from 1.3 million) 
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e One percent increase in enrollment of students who are of two or 
more races (+5,000 students). 


Additionally, the projections suggest decreased enrollments of white 
students (6% decrease in enrollments of white students: 9.9 million in 
2028, down from 10.5 million in 2017) and American Indian/Alaska 
Native (9% decrease, from 138,000 to 125,000 in 2028). While 2003- 
2017 saw a 17% increase in growth (16.9 million to 19.8 million) in 
students attending postsecondary institutions, from 2017 to 2028, the 
data projections indicate only a 3% increase, up to 20.3 million. The 
demographic data suggest that the increased population we will see in 
the coming decade is more diverse that those cohorts who have come 
before in terms of race and ethnicity. These changes impact our ability 
to generalize and provide transferable findings from research. 

Beyond education specific data on demographic trends, the U.S. 
Census also suggests that nationally our population will continue to 
change. The demographic shifts occurring outside of specific educa- 
tional contexts will impact not only our settings in higher education, 
but also the applicability of research across spaces, communities, and 
populations. Our participant populations should adequately represent 
the populations the findings are intended to serve. 


Methodological Reflections 


We look for what we want to learn. We seek to understand in ways con- 
sistent with our worldview and our implicit assumptions about the pop- 
ulations to which we intend to generalize. For me, the most compelling 
sort of data is census data because it is an alternative to sampled data 
and thereby avoids important types of statistical bias. By census data, I 
mean the inclusion of 100% of data. This sort of data collection with 
human participants is notoriously difficult. A problem with census data 
is it can take a long time to collect, usefully record, and a tremendous 
amount of infrastructure and time to analyze. It’s also not necessary to 
use census data if you are comfortable using an adequate sample. For 
instance, in Writing Studies, Card’s (2020) article is a great example of 
how to effectively use a random sample to make claims that are general- 
izable with a reasonable margin of error and confidence level. 

This particular collection of census data illustrates the precarious na- 
ture of IRBs’ workloads, and this informs how we engage with IRBs. 
Many studies an IRB reviews will fail to meet researcher self-stated 
enrollment thresholds. The National Center for Education Statistics 
(NCES) data (de Brey et al., 2019; Hussar & Bailey, 2020) also show 
that the majority of students in our classrooms in the next 20 years will 
look so vastly and beautifully different from our current student pop- 
ulation. At present, if we maintain status quo, we will be ill-prepared 
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to serve our future students and evaluate and improve the learning en- 
vironments we care so deeply about. With these matters at the fore, in 
Chapter 4, I use the data shared in this interchapter to argue that a 
collaborative approach to IRBs and federal policy is now an imperative. 
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4 Don’t Be Too WEIRD 


Research for the Future 
of Writing Studies 


Institutional Review Boards (IRBs) are keenly attuned to local research 
contexts and communities. IRB members and staff can support Writing 
Studies researchers in the design of successful studies. Because IRBs are 
privy to researchers’ study development, review, implementation, and 
closure processes, they have considerable insight to share. In this chapter 
I develop the following argument: 


Designing research for the future of the field demands researchers 
attend to the nuanced work of developing effective recruitment 
strategies and diversifying participant populations within feasible 
study designs that promote justice-driven results. 


I develop this argument in three interrelated sections: 


e The first section, “Are We Just (Re)Producing WEIRD Stuff?” offers 
an elaboration and brief histories of key terms including WEIRD 
populations, recruitment, and reproducibility. 

e In the second section, “Implications of Two IRB’s Closure Meta- 
data” data from two IRBs illustrate how researchers’ recruitment 
and retention strategies impact researchers’ experiences with IRB 
review. 

e The third section, “Research Design for the Future of the Field” is a 
reflection on our recruitment and retention behaviors as we design 
research for the future of the field. 


Methodological frames developed and/or traditionally used in social sci- 
ence disciplines now circulate more widely in Writing Studies. Quantita- 
tive work, seen often in the very early years of the field (Poe, 2019) but 
less frequently since (Haswell, 2005) is now emerging as a crucial strand 
of considered inquiry. Alongside our increasingly multidisciplinary 
work, Writing Studies researchers are grappling with the attendant 
concerns of reproducibility of research and the diversity of participant 
populations. Both of these, as well as the engagement in empirical prac- 
tice, are ethical matters. A heretofore implied, and now overt, premise of 
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this book is that there is an ethical obligation to conduct more empirical 
work with participants in Writing Studies. Elliot, in his 2016 theory of 
ethics for writing assessment, made plain the ethical imperative to prac- 
tice empirical work (see, for instance, p. 221 and § 3.3.4). Outside of and 
beyond writing assessment work, the field of Writing Studies is one that 
impacts students, teachers, writing programs, and communities beyond 
the university. Given the consequences, dismissing efficacious empiri- 
cal practices can be unjust. Reproducibility of research, a consideration 
for empirical researcher, is also an ethical concern. Reproducibility sug- 
gests that participants’ contributions are used ethically through verified 
findings or that participants’ contributions help retheorize a particular 
issue—if a study cannot be reproduced—to generate new strategies for 
addressing a social issue. 

The ethical concerns of reproduction also are challenged by the lack 
of diversity in the populations under study. These issues correspond with 
our obligation to serve increasingly diverse populations with verified 
interventions. Research which enrolls a monoculture of participants, 
largely from a WEIRD (Western, educated, industrialized, rich, dem- 
ocratic) populations can potentially not be reproducible when applied 
to diverse groups. These concerns remind us that highly contextualized 
research is not often useful for broad implementation for pragmatic 
purposes, or replication and/or reproduction for verification purposes. 
Imperative, then, is our recognition of the impact research can and will 
have, and how this impact contributes to the public good. 

Beyond the work we can do as researchers, it is important to under- 
stand that IRB representatives and researchers who responded to the 
RAND-NIH Survey introduced in Chapter 2 are overwhelmingly white 
(91.1% of chairs, 86.8% of members, and 74.3% of Principal Investiga- 
tors [PIs]). This raises issues related to increasing diversity and equity in 
board membership and representation in research by minorities (Berry 
et al., 2019, p. 46). The whiteness of our local IRBs and the demograph- 
ics of the researchers with whom IRBs work is a stark counterpoint to 
the populations we enroll to ensure our findings are useful across con- 
texts and contribute to foundations for programs of study upon which 
the field can build—or, put more simply, generalize. Many researchers 
do not believe it within the IRB’s purview to promote diversity in study 
populations (65% of respondents either disagreed or strongly disagreed 
this was the purview of an IRB [Berry et al., 2019]). And yet by enrolling 
WEIRD participants, and, among these populations, even more histori- 
cally hegemonic demographics, researchers perpetuate findings that are 
not broadly applicable to other contexts and/or populations. 

This chapter aims to strike a careful balance among reproducibility, 
generalizability, and the foundational ethical principles of autonomy, 
beneficence, and justice, and takes for granted that we have an ethical 
imperative to conduct empirical research in contexts where findings are 
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consequential for communities. All of these issues contribute in import- 
ant ways as we work to increase the diversity of participants and design 
for reproducibility. To address these concerns, we can (1) diversify our 
participant populations, such that findings from research will provide 
successful interventions and innovations for the future of the field and 
(2) build and share robust research design and corresponding datasets 
in public forums in efforts to build a field more able to answer calls for 
reproducibility and transparency. 


Are We Just (Re)producing WEIRD Stuff? 


WEIRD-ness in research is an essential concept to understand because 
it informs national conversations about research ethics and research 
design, which further influence IRB review of research and our experi- 
ence proceeding through IRB review. To understand why—beyond our 
field’s desire to build better research for the populations we are certain 
to be supporting in the coming decades—let’s take a wider and more 
comprehensive view of some concerns invigorating researchers work- 
ing with participants in the U.S. Many of these considerations are over 
a decade old—e.g. Henrich et al. (2010) and Maxwell (2004). Noting 
that these constructs influence the same IRBs that review our work, and 
influence IRBs’ understanding of the generalizability and the principle 
of beneficence weighed with potential risk, we can recognize strategies 
for situating our work as distinct from, or perhaps similar to, these types 
of projects. 


WEIRD Populations 


In 2010 Henrich et al.’s argument about WEIRD populations—that 
the vast majority of research findings are based on data collected from 
participants who are from WEIRD cultures (Western, educated, indus- 
trialized, rich, and democratic)—made waves in 2010. The opinion, 
published in Nature, suggested that this “common” population, widely 
accessible by researchers (who themselves are often also WEIRD) does 
not always generate findings that can be generalized across contexts. In 
1994, the National Institutes of Health (NIH) articulated researchers’ 
ethical duty to ensure their participant population was diverse (NIH 
Policy, 2001). The NIH requires that recruitment efforts and enrollment 
include women and representation of participants from all populations 
to which findings are intended to impact. This effort was a response, 
in part, to the recognition that in the all too recent past, marginalized 
populations were often disproportionately asked to carry the burden of 
largely dangerous research participation without any prospect of direct 
benefit—a direct violation of the principle of justice articulated in the 
Belmont Report and manifested in the Common Rule—which requires 
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that individuals who incur the burdens of participating in research 
should also benefit from the findings. 

We are certainly caught between the context-specific nature of our 
work and our need to provide generalizable support to colleagues and 
communities. And no, we are certainly not generalizing, as some fields 
claim to be, to the whole species (Henrich et al., 2010). The concerns 
about the usefulness of our findings and the applicability of our interven- 
tions are contingent upon our context. We do context-specific research. 
In many instances when designed and implemented with justice-driven 
goals, this work can also be generalized within our field—and to broader 
institutional and demographic contexts—to support communities of 
practice and purpose, even if our populations are, for the most part, 
WERID. 


Reproducibility 


I use the term reproducible, rather than replicable, after drawing upon 
several conversations—from scholars such as Stevens (2017), Schloss 
(2018), Resnick and Shamoo (2016), and the Replicability Research 
Group (2020). When I use the term reproducible I mean that multiple re- 
searchers can arrive at similar results with similar tools and/or shared or 
extant data; this is not dissimilar to Haswell’s representation of the “R” 
in replicable, aggregable, and data-driven (RAD) research in his 2005 
article. However, unlike replicability, reproducibility does not demand 
that sufficiently similar conditions or tools be used to achieve exactingly 
similar results in the multi-site practice of research. Rather, research 
that is reproducible allows for researchers to use similar methodologies, 
participants, and analyses to generate results to substantiate, or refute, 
prior findings. To draw a quick parallel to human biology—humans can 
reproduce offspring that are genetically similar to them, but humans do 
not replicate themselves. 

Our discipline’s understanding of these terms is usefully informed by 
the U.S. Department of Education’s guidelines on replication and repro- 
ducibility work in research (Companion, 2018), which uses reproducibil- 
ity to refer specifically to extant data (which I broaden in the definition 
offered above) but also distinguishes between conceptual and direct rep- 
lication. Conceptual replication in this text is similar to reproducibility 
as I’ve discussed it here, but not practiced on extant data, necessarily. 
For the purposes of this project, I elide conceptual replication and repro- 
duction, preferring the term reproduction to replication. 


Why Reproducibility and Not Replicability? 


In Writing Studies research, reproducibility is a feasible goal and, I argue, 
something we can easily build into our study design; we can generally 
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examine interventions not significantly impacted by time across contexts 
with similar tools and populations. Replicability, or the replication of 
results based on replicated contexts, however, is not feasible for the ma- 
jority of Writing Studies research contexts, at present. To replicate—or 
run an identical study in identical contexts in hopes of finding exact 
replicas of results—is not easily done in most disciplines and is often 
best suited only for meta-analyses. When we are prepared to apply some 
best practices, such as the “replication recipe” by Brandt et al. (2014), 
replication may be a more achievable goal for the field. 

At present, Writing Studies’ first step can be to practice with repro- 
duction, unless we are working with extant datasets toward replica- 
tion. When we are collecting new data to address a research question 
addressed in another context, it can be near impossible to re-create ex- 
act contexts with participants; researchers simply do not have that sort 
of control over any research context. This suggests that reproduction, 
rather than replication, may be a more useful goal. 

While reproducibility can be a feasible goal in many of our research 
settings, the Open Science Collaboration (2015) highlighted that re- 
searchers don’t often pursue reproducibility because it is not incentivized. 
But they argue progress relies on both innovation (which is prioritized) 
and replication/reproduction work, because the latter both increases cer- 
tainty and promotes innovation. Some researchers have suggested that 
since replication is not always feasible, metascience can begin to help us 
address these issues, too (Schooler, 2014). 


Should We Only Do Reproducible Work? 


No. I am not suggesting that we should begin reproducibility work on 
all research. Nor should we pursue only reproducible work. However, 
as we scale research and realize the implications of broader programs of 
study, we can work with our local IRBs to better understand how these 
connected issues influence the research enterprise more broadly. 


A Brief Note about Pre-Registration, Study Databases, 
and Meta-analyses 


Over the past ten years, our colleagues in social science disciplines 
have prompted researchers to build for replicability and reproducibil- 
ity (Pashlers & Wagenmakers, 2012; Schooler, 2014). Alongside rec- 
ommendations and, in some cases, requirements for pre-registration 
databases, updates to federal policy encourage us to consider strate- 
gic methods to build for reproducibility in our own discipline. Other 
changes in the shifting culture of research ethics include the calls to 
build discipline-specific, pre-study databases that outline the processes 
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and expectations for a given study (Resnick, 2016). This prevents data 
cherry-picking and promotes data sharing in ways that build more 
robust tools for connecting research agendas, honing in on timely 
questions, and revisiting long-standing “truths” to reevaluate claims. 
Alongside calls for use of tools that afford opportunities for big data 
collection and analysis, we, too, should consider disciplinary meth- 
ods for consolidating findings and offering up datasets for validation 
and reproducibility. Renick notes that the goal of this sort of work is 
not intended to denigrate existing work or shame researchers. Rather, 
through this work we can better build knowledge by understanding 
effective methodological structures, questioning paradigms, and assess- 
ing how results are imbricated in findings. 

And so, while our goal is not necessarily reproducibility in all 
research, we can be more careful when using language regarding gener- 
alizability and drawing conclusions. Establishing a network of qualified 
claims based on data begins with determining our sample and recruit- 
ing participants to ensure we are adequately representing the diversity 
of communities the work is intended to serve. Additionally, when we 
prepare findings for distribution and publication, attending to WAC’s 
principles (Barnes et al., 2020) regarding sharing methods and context, 
toward the generalizability and transferability of research, is useful as 
the field builds programs of study. All of this work is enhanced by both 
methodological justice, or orienting our research around outputs and 
justice-driven goals, and methodological transparency throughout the 
research process. 


Recruitment 


Sugden and Moulson (2015) argue that recruitment is key to success in 
research because recruitment determines the final enrolled sample size, 
which is a vital consideration in its own right. The final sample of a 
study determines the generalizability of results and findings. Sugden and 
Moulson noted that differences in culture, race, and ethnicity all impact 
prospective participant behavior, which are influenced, at the outset, by 
prospective participants’ experience of the recruitment process. In using 
recruitment variability to diversify their enrolled participants, Sugden 
and Moulson leveraged tailored recruitment materials to increase the 
diversity of participants. Yet research suggests recruitment is not priori- 
tized by senior researchers (Patel et al., 2003) despite the understanding 
that the better experienced a recruiter is, the more likely hard-to-recruit 
groups will consider participating (Yancey et al., 2006), especially when 
informed strategies are considered in recruitment (Quintana et al., 
2006). Scholars have also suggested it is imperative that recruiters are 
highly motivated (Fletcher & Hunter, 2003), which can be difficult for 
large and/or diffused study teams. 
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This empirical evidence, alongside publications in Writing Studies 
examining the importance of the recruitment and consent processes 
(Pigozzi, 2017; Wright, 2012), suggest it is reasonable to engage in 
robust, participatory, collaborative and context-specific strategies for 
determining effective recruitment strategies. In order to effectively re- 
cruit, researchers need to know the research population well enough to 
engage them effectively and without coercion. Front-ending study design 
with, at minimum, pilot studies or considered engagement with commu- 
nities of inquiry, are effective means to understand how “small changes” 
frame prospective participants’ engagement with recruitment messages 
(Sugden & Moulson, 2015). Ultimately, this sort of pre-study design 
work can helpfully inform our study design, and require us to elaborate 
often unspoken expectations regarding the research. 

Tailoring our recruitment methods to involve diverse populations is 
vital as the demographics of our students and communities continue 
to shift and change in rapid and significant ways. While it is likely 
that enrollments will continue to fall among “traditional” students, 
given the low birth rate among this demographic in the U.S. after 
the Great Recession, National Center for Education Statistics (NCES) 
data illustrate the exciting changes influencing communities of prac- 
tice and purpose in higher education. These changes are but a micro- 
cosm of the shifting demographic trends in the U.S. Should we remain 
entrenched in researching a traditional student body, we do a tremen- 
dous disservice to those we intend to and are intended to serve. At the 
same time, we should be cognizant of the issue of over-researching 
and tokenizing traditionally marginalized populations without offer- 
ing direct benefit to individual participants or significant benefits to 
the community. 

Developing a keen awareness of the limitations and affordances of our 
traditional samples can provide space for reflection, especially when we 
engage in novel study design, use new-to-us methods, or embrace meth- 
odologies with which we are not thoroughly familiar. IRBs can help us 
situate this work within broader research initiatives in our local commu- 
nities and also bring their expertise as street-level bureaucrats to bear. 
Since the professionalization of IRBs in the early aughts in response to 
an era of hypercompliance (Babb, 2020), IRB personnel and members 
are often immersed in national conversations about burgeoning research 
strategies and ongoing challenges faced by researchers; these discussions 
are facilitated, for instance, by national organizations such as the Public 
Responsibility In Medicine and Research (PRIM&R) group and other 
grassroots platforms such as the Flexibility Coalition established by the 
University of Southern California. Given these initiatives and profession- 
alization of the work conducted by IRBs, along with the sheer volume of 


Research for the Future of Writing Studies 17 


research reviewed by any given IRB, IRBs are excellent resources as we 
look to strengthen our research designs and expeditiously get research 
underway. Expediency, however, does not come without cost. In the 
next section, Pll highlight issues of time to and for review by IRBs with 
data surrounding the issues of recruitment and retention of participants. 
These data suggest that regardless of the WEIRD-ness of our partici- 
pant populations, we should be cognizant of structural limitations in 
our study design; IRBs can help us identify and address such limitations. 


Implications of Two IRBs’ Closure Metadata 


The data from IRBs #1 and #2 introduced in the Fourth Interchapter 
suggest that some studies are challenged to enroll enough participants to 
make generalizable claims, which make reproducibility nearly impossible. 
There are two interconnected issues here. First, studies that under-enroll 
based on researcher-stated, methodologically specific recruitment tar- 
gets cannot, by researcher-defined criteria, produce generalizable knowl- 
edge. This means that participants were recruited and inconvenienced to 
participate in a study even though the data they shared is not of use for 
the particular questions under study. Second, under-enrollment and/or 
study failure should concern us because the number of protocols an IRB 
reviews, especially the higher volume exempt and expedited protocols, 
can result in delays for all other well-designed research that can offer 
benefits to society. 

I want to focus first on the studies that reported enrollments of 0, 
meaning no participants were enrolled and the study was, for whatever 
reason, cancelled and unsuccessful. Both IRBs had studies that closed 
and some of these closures were due to failed funding (e.g. when an IRB 
provided a just-in-time review for, say, NSF funding but the funding was 
not provided). In both the surveys discussed in Chapter 2 65% of re- 
searchers whose research required IRB review had experienced delay of 
research due to review at least once. These delays may occur for a number 
of reasons. But consider: one efficient, well-staffed IRB closed 1,720 pro- 
tocols that had enrolled at least one participant during a given timeframe. 
Concurrently, the IRB was closing protocols that did not enroll a single 
participant: an additional 804 protocols. This means the IRB had a ratio 
of two approved studies that ran—though, in more than half the cases, 
without generating enough data to generalize—to each study that was 
reviewed, approved, and closed without any recruitment whatsoever. 


Incommensurate Complaints: Time-to-Review and 
Mission Creep 


Beyond explaining some delays in the time to review, this data also di- 
rectly addresses the concern generally referred to as IRB mission creep. 
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IRB critics suggest that IRBs should not be concerned with issues of 
methodology, structural soundness, or feasibility (Gunsalus et al., 2007; 
Schrag, 2010; White, 2007). Yet inattentiveness to these issues can result 
in a markedly high numbers of studies that are cancelled or fail because 
researchers and IRB reviewers do not perform the duties of analyzing 
feasibility. IRBs able to take the mission of participant protection to be 
part and parcel with feasibility, methodological soundness, and general- 
izability are able to better support researchers in pursuit of higher overall 
enrollment rate and lower percentage of studies that under-enroll. While 
there are any number of reasons for study failure, for IRBs to invest the 
energies and time in studies that do not run is a deeply problematic issue 
in relation to time and fiscal economies, to say nothing of the just-ness 
of the research itself. 

Regarding the percent enrollment rate for each IRB, which was around 
45% at IRB #2 and between 76% and 95% (accounting for studies with 
a stated enrollment range) at IRB #1: I believe the contrast between the 
smaller, primarily socio-behavioral IRB and the larger, multi-board IRB 
at a research university with a hospital and medical center, provide inter- 
esting areas of discussion for Writing Studies scholars. IRBs have access 
to tremendous amounts of metadata; whether or not it is easily accessible 
is another issue. Many scholars have called for more empirical research 
on IRBs (Sieber & Tolich, 2015). Folx complain about IRBs, like IRB #2, 
for issues related both to mission creep and time-to-approval. What the 
data above suggests is that we cannot demand both that IRBs approve 
rapidly and maintain a strict mission dissociated from methodological 
and feasibility concerns. For when we demand this, IRBs could end up 
reviewing a ratio of 1:1:1: for each successful study, a second study en- 
rolled participants but not sufficiently to generate generalizable claims, 
and a third study was reviewed, approved, and enrolled no participants. 
For IRB #2, for every approved study that provided generalizable infor- 
mation, two did not. 


Why Does the IRB Need Expected Enrollment Data, Anyway? 


At this point, the question of why we have to justify sample sizes in qual- 
itative methodologies may have piqued your interest. What if a large 
percentage of these studies resulted in publications, even though they 
were under-enrolled? And if they weren’t published, what, aside from 
longer times to review, does it matter to Writing Studies researchers? 
Why does it matter that we establish sample sizes before we begin our 
research? 

Of the 98 studies that were closed by IRB #1, researchers estimated 
they would need between 15,794 and 16,606 participants to meet the 
demands for generalizability in their studies. All together, they enrolled 
12,712 participants over the course of their studies. Simply by the 
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numbers, this means that researchers enrolled somewhere between 76% 
and 95% of their total estimated population needed for generalizability. 
Recall from the Fourth Interchapter discussion that I removed any stud- 
ies that had a range of participants for IRB #2, given the sheer volume 
of data. Therefore, IRB #2, an IRB with a wider scope of review, had a 
44.5% enrollment rate among all studies with a stable expected enroll- 
ment number and a reported final closure number. 

Despite the diverse ways to calculate a minimum enrollment needed 
to generalize to broader populations, these data suggest studies gener- 
ally were under-enrolled at closure. These data were self-reported by re- 
searchers. By this I mean: researchers designed and submitted protocols 
to the IRB with sample sizes determined not by a specific positivistic 
framework or sample size calculator, but rather samples based on re- 
searchers’ own methodological determinants to establish generalization 
inferences. It is worth noting again that IRBs are keenly aware that some 
research is intended to generalize widely across time or space to a small 
group of people. Other research may generalize to a large population in 
a specific geographic location. The generalization inferences of research 
depend upon the sample (and whether the sample was sufficient) and the 
data collected (and if enough data was collected) to make generalizable 
claims. Functionally, as exhibited by the fact that these estimates were 
just that—estimates—and were established by researchers, there is no 
battle to be fought about what a sample size should be. Instead, IRBs 
prefer for researchers to justify their sample size. Therefore, there are 
no constraints on the paradigms or methodologies that can easily be 
reviewed by IRBs. Rather, judicious researchers can adequately antici- 
pate how many participants they’ll work with to collect sufficient data 
within any given methodology. For qualitative work, where information 
about the required population size is not apparent in the early stages 
of research, researchers can rely on existing scholarship using similar 
methods to determine thresholds of participation. IRBs can help here, 
too, as they may have reviewed scholarship from similar methodological 
frameworks and can put researchers in touch with one and other about 
these matters. 

IRBs understand that different paradigms and methodologies place 
demands on researchers to determine an adequate sample size. IRBs 
ask for enrollment information for two major reasons. First, so that re- 
viewers can determine whether findings will be generalizable. A second 
reason is to ensure researchers are attending to the issue of social benef- 
icence. So, for IRBs #1 and #2, thousands of participants were enrolled 
in research that was not, by definition, research. The findings from these 
studies could not be generalized. And yet, per the expedited review level, 
these studies were considered an investigation that, by its very existence, 
encroached on participants’ autonomy, and presented some degree of 
risk, if even only minimal. 
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Recruitment toward Generalization as a Cornerstone of 
Ethical Research 


We are a field that claims to generalize to communities of practice, 
purpose, and process to fulfill personal, departmental, institutional, and 
disciplinary missions across communities. With an increasingly global 
community and inclusive higher education environment, we are compelled 
to face the challenges of effective recruitment and retention of diverse 
communities. The effective recruitment of participants is a cornerstone 
of successful study planning and is often overshadowed by concerns re- 
lated to consent, enrollment, and participation. Recruitment determines 
who is contacted, what participants’ response rates are likely to be, and 
how likely the sample actually represents the population that the data 
is intended to generalize to (see, for instance, Bornstein et al., 2013). 
Moreover, recruitment, when done poorly or with inadequate prepara- 
tion and implementation, can systematically exclude certain communi- 
ties and overburden others (see, for instance, Clark, 2008; Sukarieh & 
Tannock, 2013). If there are, already, issues at the meta-level simply with 
recruiting enough participants (per the data from IRBs #1 and #2), the 
ability to produce generalizable information based on WEIRD samples, 
or samples of convenience, is predictably more problematic. 

Our ability to design principled and transparent frameworks that will, 
in turn, allow qualified generalization inferences and research reproduc- 
tion are informed by these issues of recruitment, retention, and participant 
diversity. For Writing Studies research that requires and receives IRB re- 
view, the approval letter does not suggest any level of rigor or a “rightness” 
of methods; indeed, IRBs are tasked to ensure the safety of participants 
(Whitney, 2016), not to police methods. That said, some IRBs do incor- 
porate the work of methodological examination into review. These IRBs 
believe that merely approaching prospective participants is to encroach on 
their autonomy. There is always something lost, and not necessarily al- 
ways something gained, when prospective participants are approached by 
researchers yet the research does not produce generalizable findings. Spe- 
cifically, it is detrimental to the broader research enterprise and the public’s 
trust in research and researchers—I discuss this further in Chapter 6. 

What I am suggesting here, then, is that researchers’ critiques often set 
up IRBs to fail. Researchers cannot expect IRBs to not address issues of 
methodology and feasibility and at the same time complain about time 
to review for any sort of study we may want to pursue. It is here I see a 
tremendous opportunity for researchers to work on the meta-analyses 
of IRBs, both outside and inside of Writing Studies. Because some IRBs’ 
systems are not conducive to empirical meta-research, and we similarly 
lack robust frameworks for disciplinary meta-analyses in Writing Stud- 
ies, I see similarities between Writing Studies and IRBs. There is a cor- 
relation between the lack of empirical data on IRBs and empirical data in 
Writing Studies, which results in critique to both. Unlike IRBs, however, 
Writing Studies’ sustained existence is not required by federal policy. 
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When we more clearly articulate our paradigms, methodologies, and 
methods, the infrastructure that enlivens our work—particularly 
IRBs—can be realized as, rather than a barrier, instead as partners and 
allies who offer constructive feedback and engage with active interest in 
our work. From my experience in many roles, IRBs invite this critical 
work and assign it great value. IRBs demand considered attention to 
what research will provide to the public. IRBs endeavor, when read- 
ing protocols, to understand the impetus behind decisions that could 
put prospective participants at risk—even if this is merely a researcher 
bothering them, asking if they are interested in participating in research. 
With such a mindset, IRBs serve as pilot tests for publics and provide 
scrutiny outside of our discipline that help us strengthen work on behalf 
of our communities. 

Regardless of our methodological and paradigmatic orientation—and 
any perceived (or actual) incommensurability among these within our 
field—IRB review is concordant with our goals of honoring individu- 
als’ autonomy, demanding beneficent inquiry, and seeking justice. As 
the field grows, changes, and adapts to the new communities and demo- 
graphics it serves, we can be thoughtful about creating deliberate study 
designs that emphasize diversity in recruitment and retention. A keen 
understanding of the impact of research for participants and future ben- 
eficiaries needs to be effectively articulated. Because of their bird’s-eye 
view of local research contexts, IRBs can help us attend to these issues. 


Accounting for Crucial Methodological Shifts in 
Writing Studies 


It is crucial at this juncture to engage with the concerns circulating in 
our scholarship regarding the value of dismantling systematic oppres- 
sion and creating new opportunity structures working both within and 
outside of our current research infrastructure—similar to the differen- 
tiation of ideal v. conservative justice described on p. 41. Because IRBs 
are part of existing infrastructure, I do acknowledge their utility may 
be fraught for some of our scholars on the frontlines of revisioning re- 
search for the future of the field. However, I see no incommensurability 
between these two approaches to justice. They are commensurate if they 
are oriented toward a similar vision of justice. One approach pursues 
justice via dismantling, wholesale, the predominant narratives around 
research. This is work done “with-out,” and from outside, the dominant 
paradigm. The other approach seeks justice “with-in,” by deploying ex- 
isting tools within dominant paradigms to illustrate infrastructural in- 
justice. Both strategies can be effective toward different ends. 

Working with IRBs, and subscribing to the frameworks that require 
their review, is not without critique among methodologists both inside 
and outside Writing Studies. For instance, Walton et al. note in their 
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2019 book, subscribing to dominant, traditional models of inquiry can 
result in the continued marginalization of communities who have been 
already systematically disadvantaged by our current infrastructure. 
Similarly, Tuck and Guishard (2013), Patel (2015), and Tuhiwai Smith’s 
(2013) vital decolonial scholarship question how researchers allow IRBs 
to “flatten” work with participants in many ways, including through 
the use of standardized informed consent templates, for instance (Patel, 
2015, p. 15). Patel usefully draws upon the settler colonialist deference 
for ownership and expands this in discussions of data ownership and 
how this is mediated by IRB review (p. 36). These critiques prompt us to 
consider how we reinscribe settler colonialism through our compliance, 
and whether there are negotiable components in our work. 

My work here to encourage folx to recognize the affordances of meth- 
odological diversity and understand the implications of sampling plans, 
generalizability, and retention are tied to the concrete reality of the econ- 
omies in which we circulate, the transparency we wish to have in our 
work, and the necessity of engaging carefully with assigning general- 
izability claims to individual experience. Participating and engaging in 
disruptive models affords opportunity to begin dismantling marginaliz- 
ing behaviors. Yet at the same time, to continue our work, we will neces- 
sarily need to attend to the constructs that maintain our ability to do so. 

Even those methodologies and methods to which we look for disman- 
tling the hegemonic discourses surrounding research are benefited by ex- 
amining how confirmable they are in extended settings, how replicable 
they are in similar contexts, and how verifiable they are with individuals 
in corollary situations. Gomes (2018) wonders if “our work might be 
more impactful if more of us conscientiously adopted replicable, aggre- 
gable, and data-driven (RAD) approaches to social justice work.” (402). 
His call, in a collection on social justice and writing assessment, im- 
plicitly recognizes the challenges of both dismantling something broken 
to build something new—with old tools. These debates are evergreen 
in our scholarship. Yet IRBs function outside of these disciplinary dis- 
cussions and can lend a useful ear to researchers troubling with which 
strategy is best suited for participants. 


Recalling the Roots of IRBs: Building Better from With-in 
and With-out 


Regardless of paradigmatic allegiance, IRBs exist because of the consid- 
erable injustices brought upon communities who, to this day, continue to 
be disadvantaged and marginalized. The praxis of research ethics over- 
sight demands tangible, actionable policy to protect those most vulnera- 
ble to further marginalization. I appeal, therefore, for those interested in 
critiquing IRBs to seriously consider why IRBs would impede the work 
we aim to do, recalling the epigraph from Pelmon’s at the start of this 
book: “The IRB is not the sinister adversary of the virtuous scholar” 
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(2019, p. 82). From a policy and practice standpoint, IRBs exist for a 
very clear reason—to honor participants’ autonomy, demand beneficent 
inquiry, and seek justice within the research enterprise. IRBs fill a gap 
in our social system, and are a check for researchers who are necessarily 
self-interested in the success and implementation of their research. The 
goals of IRBs resonate clearly with our own goals as a discipline, regard- 
less of whether we aim to dismantle the system from without by way of 
new tools or from within. And moreover, IRBs are helpful partners for 
us as we recognize and act on the ethical imperative to conduct empirical 
work. 

In Chapters 2 and 3, I established IRBs as participant advocates keenly 
trained to attend to issues of autonomy, beneficence, and justice. In this 
chapter, I’ve further illustrated that because of their bird’s-eye view of 
our local research contexts and considerable experience reviewing and 
observing the realities of research, IRB members and staff are partic- 
ularly well-equipped to support researchers. Whether they work with 
us in initial preparations, point out useful strategies for recruitment or 
retention, direct us to colleagues using similar tactics, or offer us insight 
into best practices for novel methods, IRBs are helpful collaborators in 
the research design process. 

The datasets from two IRBs, alongside broader demographics trends 
and changes in communities, offer an impetus to design with partici- 
pants for study success. As we design research for the future of the field, 
we can critically consider how we can diversify research populations so 
they most accurately represent the populations we aim to better serve. 
Moreover, we cannot further burden already marginalized populations. 
IRBs and research communities are essential partners in this process. 
Given Writing Studies’ keen attunement to the reality that all research is 
context-specific, we should be able to clearly articulate a direct benefit 
to prospective participants or a tangible benefit to a broader community. 
In Writing Studies research, one of these two outcomes is necessary to 
fulfill the criteria of beneficence in research with human participants. 


Methodological Justice and Transparency 


What does this have to do with research for the future of the field? It 
depends entirely on what you believe the future of the field should be. 
I can suggest that these considerations, and engagement with IRBs, 
afford us a few advantages in the coming decades, as the results of 
the human protections policies in the U.S. are assessed, perhaps re- 
vised, and broadly implemented. First, with these issues in mind, we 
can address and understand the workloads and realities that IRBs face 
on a day-to-day basis. We can begin to understand how our work, 
while crucial to us, is one small piece of their function. Second, we 
can position our work and its broader application and generalizability 
within our local contexts. We can acknowledge the populations who 
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have been traditionally marginalized, ignored, or heavily recruited 
and, with this information, work with our local IRBs to ensure our 
prospective participant population is both diverse and large enough 
to help us answer research questions. Finally, IRBs are able to usefully 
observe the research enterprise in ways individual researchers cannot 
and with the added benefit of considerable remove. This makes them 
uniquely qualified to be frank with us about what we propose as bene- 
fits of research; these challenges are not, however, limitations. Instead, 
they are invitations for us to more fully share our vision for why the 
work is necessary. In such instances, methodological justice and the 
persuasive articulation of axiological principles are key to collabora- 
tive communication. Such an asset-based approach, which includes 
transparency about research goals in consideration of what is needed 
from participants, can all help researchers communicate more effec- 
tively with their IRBs. 


But Is It OK To Be WEIRD? 


So yes, it’s OK to be WEIRD. Part and parcel with un-WEIRDing our 
samples and enrolling for generalizability is building for reproducibil- 
ity. The NCES data on future enrollments in colleges and universities 
is unequivocal. National census data is unequivocal. For the benefit of 
communities, engagement in more reflective work is necessary. These 
considerations relate to who we enroll in research, how we recruit them, 
and whose and what ends that research serves. 

The longevity of our discipline relies on our ability to adapt to novel 
environments and populations and offer transferable findings for im- 
plementation across contexts. We can offer novel methodologies and 
use novel methods, but if our selection and enrollment criteria remain 
staid in the late 20th century, we run the risk of missing a paradigm 
shift. We can use traditional and new tools to interrogate, problema- 
tize, and evolve research practices to serve the public good. This work 
should always carry the hallmarks of justice-driven research, which im- 
bricates all aforementioned considerations of reproduction, recruitment, 
and retention with keen attention to appropriately diversifying study 
populations—and practicing transparency in our publications about 
these issues and how we approached them through research design. 


Summary 


IRBs demand excellence of researchers on behalf of participants. 
Recognizing the many demands on IRBs, we can more effectively and 
efficiently get research underway if we articulate clearly to our local 
IRBs what we need to find out and why. This chapter broached issues af- 
fecting research with human participants across disciplines, paradigms, 
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contexts, and sites of inquiry. I focused here on two major issues of in- 
creasing importance for Writing Studies research: diversity and adequate 
enrollment of participants. 

First, our disciplinary research would be strengthened if we continue 
to grapple with the issue of how to better represent the publics we aim 
to serve in our study populations. Second, we can also acknowledge that 
our IRB may review a number of studies that do not enroll sufficient 
participants to offer generalizable claims. The labor demanded of IRBs 
does come at a cost, including delayed time to review. For in the spirit of 
fairness and practicality and accountability, IRBs typically work on re- 
views in strategic, often chronological fashion. They are unable to triage 
based upon merit or feasibility. For these reasons and in such contexts, 
limited resources limit everyone. 

Designing research for the future of the field demands we attend to 
these complex issues with balanced consideration. Taken together, these 
issues of recruitment, retention, and generalizability suggest that the state 
of research has challenges far greater than those experienced in any one 
discipline. In the Fifth Interchapter I discuss some of the major changes 
made to the Common Rule. Then, in Chapter 5, I parse out these updates 
and build upon all the concepts I’ve discussed to this point—from Writing 
Studies researchers’ experiences and perceptions of IRBs to WEIRD pop- 
ulations to diversity to retention to generalizability to balanced review— 
and discuss major sites of research that will be impacted in the coming 
years by policy change and methodological evolution. 
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